Idea: Using simulation to prove there isn’t enough evidence in favor of sarilumab

In this document, we will elaborate on our new idea. We will display multiple plots but bear in mind most of them are shown here to help you understand the underlying details of this analysis. In the end, we will highlight what plots would be essential in a research article in my opinion.

Research question

Does sarilumab reduce mortality in hospitalized patients with COVID-19?

According to WHO’s conclusion, sarilumab reduces mortality in hospitalized patients for COVID-19. However, it is clear that this conclusion was mainly driven by tocilizumab’s effect. As shown below, sarilumab by itself showed inconclusive results. Thus, we sought to further analyze these results.

To explore both the uncertainty and the need of future research on sarilumab, we suggest that we should perform simulation with future studies. We will thoroughly explain the details below.

WHO’s meta-analysis

Although the main figures in WHO’s article show fixed-effect meta-analyses (MA), they also performed random-effect (RE) analyses for sensitivity analyses. Below, we display their eFigure 3 (RE MA for mortality).

Of note, they found an odds ratio of 1.08 (95% CI 0.83 - 1.40) for sarilumab. Because they also pooled both tocilizumab and sarilumab data, the overall effect was 0.90 (95%CI 0.76 - 1.05).

Our reanalysis

In our Bayesian reanalysis, we performed a random-effect meta-regression in which we used both sarilumab and tocilizumab data. In contrast to WHO’s model above, we estimated sarilumab’s and tocilizumab’s effect separately, as shown below:

\[ \begin{align*} y_i & \sim Normal(\theta_i, \sigma_i^2) \tag{Likelihood} \\ \theta_i & \sim Normal(\mu, \tau^2)\\ \mu &= \beta_0 + \beta_1 x\\ \\ \beta_0 & \sim \operatorname{Normal}(0, 1.5^2) \tag{Priors} \\ \beta_1 & \sim \operatorname{Normal}(0, 1^2) \\ \tau & \sim \operatorname{Half-Normal}(0.5) \\ \end{align*} \]

Here are the posterior distributions of tocilizumab’s and sarilumab’s overall effect:

Estimated posterior distributions for tocilizumab’s (0.84 [95% Crl 0.71, 1.00]) and sarilumab’s (1.06 [95% Crl 0.78, 1.37]) overall effect. For tocilizumab, there is 97% of probability of odds ratio being lower than 1, while 36% for sarilumab.

As mentioned above, we did not pool tocilizumab and sarilumab data together. Instead, we separately estimated their effect. To be able to compare to WHO’s pooled results (0.90 [95%CI 0.76 - 1.05]), we will now fit a (Bayesian) model similar to theirs:

\[ \begin{align*} y_i & \sim Normal(\theta_i, \sigma_i^2) \tag{Likelihood}\\ \theta_i & \sim Normal(\mu_{toci+sari}, \tau^2)\\ \\ \mu_{toci+sari} & \sim \operatorname{Normal}(0, 1.5^2) \tag{Priors}\\ \tau & \sim \operatorname{Half-Normal}(0.5) \\ \end{align*} \]

In this case, we can estimate the posterior distribution of tocilizumab and sarilumab pooled effect (\(\mu_{toci+sari}\)):

Estimated posterior distribution for pooled overall effect (0.89 [95%Crl 0.76 - 1.06]). There is 93% of probability of odds ratio being lower than 1.

Predicting future studies

WHO’s meta-analysis and our results above focused on the overall effect for each drug. In addition to these results, one could estimate the prediction intervals (WHO did not).

A random-effect meta-analysis assumes there are two sources of variation: \(\sigma^2\) (within-study heterogeneity) and \(\tau^2\) (between-study heterogeneity). While \(\sigma^2\) represents the know variance in each study (extracted from the original study), \(\tau^2\) represents the difference between the studies included in the meta-analysis. To estimate the prediction interval, one have to take in account the between-study heterogeneity.

Prediction intervals present with many advantages, such as helping to better understand the heterogeneity between studies and how it influences results. Moreover, one of the greatest features of prediction intervals is that they also inform probable values for the true treatment effect in future settings.

Because we performed random-effect meta-analyses, we can estimate the predictive distributions for tocilizumab, sarilumab and pooled effect:

Notably, the predictive distributions are wider than the distributions we showed before, because now we are taking in account the estimated between-study heterogeneity (\(\tau^2\)). All three distributions present with some probability of harm (odds ratio > 1), especially sarilumab’s distribution.

In addition to features mentioned above, Bayesian random-effect meta-analyses present with a specific interesting feature in regard to the predictive distribution:

• We can simulate (predict) the point estimates of future studies.

Below, I show 50 different future studies that were randomly drawn from each predictive distribution. In each plot, we display the respective predictive distribution on top (mean and 95% credible interval) and simulated point estimates below:

We hope you can appreciate that tocilizumab’s predictive distribution (PD) is narrower and its respective future studies are closer to each other. In contrast, sarilumab’s PD is wider, and thus future studies are far apart. Pooled PD yielded mixed results in this simulation.

The same phenomenon is displayed in one of Dr Williams’ article:

Cumulative meta-analyses with future studies

As shown above, we can simulate point estimates of future studies with our Bayesian models. Although these results already highlight the considerable uncertainty around sarilumab’s effect (both current and forthcoming evidence), it is of great interest to better understand the need for future research on sarilumab.

According to both WHO’s analysis and our models, it becomes clear the current lack of certainty around sarilumab’s effect. Notably, there is no evidence-based “proof” that these drugs are equivalent for COVID-19. The only justifiable rationale is biological plausibility because they are both IL-6 antagonists, even though mechanisms of action and pharmacokinetics are not the same. In fact, current evidence only shows that these drugs probably do not have the same effect. However, WHO assumed tocilizumab and sarilumab are equivalent and focused on pooled results to change their guidelines.

To further analyze the impact of different assumptions on whether these drugs are equivalent or not, we will leverage from the predictive distributions. In summary, we will generate future studies and combine with current evidence on sarilumab (9 studies). We can then quantify the impact of these assumptions over the long run and pinpoint when sarilumb will show a robust evidence of benefit (or not).

Because we wnat to combine current with “future” evidence, we will perform random-effects meta-analyses. In contrast to the models above, we will fit cumulative meta-analyses, i.e., we will add one future study by one and estimate the “new” overall effect.

To perform such analyses, we have to make a few assumptions. Predictive intervals only yield point estimates – eg, mean effect – of future studies. To fit a meta-analysis, we require both mean effect and the underlying variance for each study. Therefore, we will assume all future studies present with the same variance. Also, we will assume this variance is equal to the the average variance in current tocilizumab studies, which is 0.04 in the log odds ratio scale.

To better visualize these assumptions, we will now show the estimated 95% confidence intervals around 50 random future studies drawn from tocilizumab’s predictive interval (same as shown before). These confidence intervals are directly related to the chosen fixed variance (0.04) which yields a 95% confidence interval equal to \(+-\) 1.09 around the mean in the odds ratio scale.

We used the log odds ratio scale in this plot to maintain the visual symmetry of confidence intervals:

The plot above only shows the mean and 95% confidence intervals of 50 randomly sampled future studies from tocilizumab’s PD. Now, we will show the results of the cumulative meta-analysis based on these 50 studies. Moreover, we also show two other cumulative meta-analyses based on studies sampled from sarilumab’s and pooled’s PDs (already displayed before).

The X axis represents the estimated overall odds ratio (mean + 95%CI) for each cumulative meta-analysis, i.e. the combined effect of current sarilumab evidence with Y future studies. The Y axis depict how many studies had been included thus far each cumulative meta-analysis, from 1 to 50.

According to the plots displayed above, it is clear that when multiple simulated future studies are combined to sarilumab’s current evidence, the overall effect approximates to the mean of the underlying PD.

Of note, we only sampled 50 studies and performed one cumulative meta-analysis for each PD. Of course, these samples are random, and there is uncertainty around them. Thus, to better estimate the simulated overall effect, we repeated these analyses 50 times in respect to each PD, i.e., we performed 50 different cumulative meta-analyses (per PD) by including 50 additional studies in each of them.

We display below the resulting 95% CIs, while accounting for the difference between all cumulative meta-analyses.

Solid black lines depict the mean value, and areas depict the 95% credible intervals of all simulations combined.

Lastly, to further understand these results, we can quantify the posterior probability of any benefit (OR < 1) for each step in these cumulative meta-analyses"

Black solid lines decict the mean, and areas depict the 95% credible intervals of all simulations combined. The X axis depict how many studies had been included thus far each cumulative meta-analysis, from 1 to 50. The Y axis depic the probability of any benefit, i.e. of odds ratio lower than 1.

In conclusion, current evidence on sarilumab shows a 36% probability for any benefit in reducing mortality. If we assume sarilumab is equivalent to tocilizumab, we would require 10 future studies to reach >95% of probability for any benefit. On the other hand, if we assume sarilumab is not equivalent to tocilizumab, there would be <50% of probability that sarilumab reduces mortality, regardless of the number of studies included in a meta-analysis.

Therefore, based on current evidence, the probability of harm is greater than of benefit in regards to sarilumab. Moreover, in case we assume sarilumab is equivalent to tocilizumab, we would require several future studies to find evidence of any benefit for sarilumab.

Main Figures

Figure 1

Estimated posterior distributions for tocilizumab’s, sarilumab’s, and pooled’s overall effect.

Figure 2

Simulated posterior probabilities of any benefit (odds ratio < 1) in cumulative meta-analyses including sarilumab’s current evidence with future studies.